We have been using Zoom/Google Meet/Chime for remote lecturing, group discussion, exam proctoring... and you might already be familiar with many of its functionalities. In this final project, you will implement a simplified meeting transcription system that allows you to perform the speech diarization task with a meeting-like long recording.
Remember that we mentioned in the previous homework that the task of speech diarization is to solve who speaks what at when. After the four coding homework, we now know that the system needs at least three modules: a vocal activity detector to find out the periods that there are people speaking (when), a speaker identification module to determine the active speakers (who), and a speech recognition module to transcribe the spoken words (what). Given that we may have overlapping speech in a meeting, a speech separation module can also be necessary to deal with those segments. Fortunately you have all gained experience on the four sub-systems, and now it's time to combine all of them to build a full pipeline for this meeting transcription system.
Here you will be asked to *overfit* your system on a 2-minute recording containing 4 active speakers. Let's first take a look at the data.
import numpy as np
import librosa
import os
import time
import soundfile as sf
import matplotlib.pyplot as plt
from IPython.display import Audio
# load the audio and the label
sr = 16000
audio = np.load('audio.npy')
label = np.load('label.npy', allow_pickle=True).item()
# listen to the audio
Audio(audio, rate=sr)
# take a look at the label
label
{'Speaker 1': {'Segment 1': {'Start': 22.723978357231125,
'End': 26.819978357231125,
'Text': 'ONE WHO WRITES OF SUCH AN ERA LABOURS UNDER A TROUBLESOME DISADVANTAGE'},
'Segment 2': {'Start': 60.43879936299203,
'End': 63.76679936299203,
'Text': 'THE VERY EMPERORS HAD ARRAYED THEMSELVES ON HER SIDE'},
'Segment 3': {'Start': 81.56636802097503,
'End': 85.43836802097503,
'Text': 'BUT IF THE EMPERORS HAD BECOME CHRISTIAN THE EMPIRE HAD NOT'},
'Segment 4': {'Start': 86.05732416302824,
'End': 89.73732416302825,
'Text': 'THE MENS SANA MUST HAVE A CORPUS SANUM TO INHABIT'},
'Segment 5': {'Start': 92.7392943983688,
'End': 96.6432943983688,
'Text': 'AND THE NEW BLOOD AT THE ERA OF THIS STORY WAS AT HAND'},
'Audio': array([0., 0., 0., ..., 0., 0., 0.])},
'Speaker 2': {'Segment 1': {'Start': 8.508374731142482,
'End': 11.61237473114248,
'Text': 'SHASTA RAMBLES AND MODOC MEMORIES'},
'Segment 2': {'Start': 29.54221387427849,
'End': 34.05421387427849,
'Text': 'GO QUIETLY ALONE NO HARM WILL BEFALL YOU'},
'Segment 3': {'Start': 41.471866716935175,
'End': 43.93586671693517,
'Text': 'BUT IT IS FAR BETTER TO GO AFOOT'},
'Segment 4': {'Start': 46.49071091155697,
'End': 48.826710911556965,
'Text': 'ONLY A LITTLE FOOD WILL BE REQUIRED'},
'Segment 5': {'Start': 68.57667956990687,
'End': 73.76067956990687,
'Text': 'THE LONG GRAY SLOPES LEADING UP TO THE GLACIER SEEM REMARKABLY SMOOTH AND UNBROKEN'},
'Segment 6': {'Start': 108.65574845433495,
'End': 113.00774845433496,
'Text': 'MOUNT BREMER IS THE MOST NOTED STRONGHOLD OF THE SHEEP IN THE WHOLE SHASTA REGION'},
'Audio': array([0., 0., 0., ..., 0., 0., 0.])},
'Speaker 3': {'Segment 1': {'Start': 0,
'End': 2.688,
'Text': 'YES AND A VERY RESPECTABLE ONE'},
'Segment 2': {'Start': 9.766205921685748,
'End': 13.862205921685748,
'Text': 'THE LADY IS NOT THE MOTHER OF THE BOYS BUT THEIR AUNT'},
'Segment 3': {'Start': 14.968394324189424,
'End': 18.520394324189425,
'Text': 'THE BOYS BELONG TO THE GENTLEMAN WHO IS A WIDOWER'},
'Segment 4': {'Start': 18.958493553627832,
'End': 20.878493553627834,
'Text': 'GEORGE NODDED'},
'Segment 5': {'Start': 27.632936904883366,
'End': 31.280936904883365,
'Text': 'THE BOYS LOOK WIDE AWAKE ENOUGH IF THE FATHER DOES NOT'},
'Segment 6': {'Start': 35.201565967133256,
'End': 40.41756596713326,
'Text': 'DO THEY STILL INSIST THAT MISS CHALLONER WAS THE ONLY PERSON IN THE ROOM WITH THEM AT THIS TIME'},
'Segment 7': {'Start': 44.49118009104179,
'End': 46.18718009104179,
'Text': 'GEORGE'},
'Segment 8': {'Start': 48.132972110998956,
'End': 51.58897211099896,
'Text': 'HAVE YOU EVER THOUGHT THAT SHE MIGHT HAVE BEEN A SUICIDE'},
'Segment 9': {'Start': 52.04267006683373,
'End': 56.93867006683373,
'Text': 'I KNOW IT SOUNDS FOOLISH BUT THE ALTERNATIVE IS SO IMPROBABLE'},
'Segment 10': {'Start': 58.1907612627863,
'End': 63.2467612627863,
'Text': 'I INQUIRED OF GEORGE WITH MY EYES STILL ON THIS FURTIVE WATCHER'},
'Segment 11': {'Start': 64.51255797049647,
'End': 67.96855797049648,
'Text': 'I TOOK QUITE A FANCY TO HIM WHY'},
'Segment 12': {'Start': 72.46209710318686,
'End': 75.11809710318687,
'Text': 'I AM LOOKING AT HIM NOW'},
'Segment 13': {'Start': 79.11250964096247,
'End': 82.95250964096248,
'Text': "YES HE'S MERCURIAL IN ALL HIS MOVEMENTS"},
'Segment 14': {'Start': 98.42025900367739,
'End': 99.89225900367738,
'Text': 'WHAT DOES HE WANT'},
'Segment 15': {'Start': 103.94549535455185,
'End': 107.65749535455186,
'Text': 'I ASKED AS SOON AS GEORGE HAD RETURNED TO MY SIDE'},
'Segment 16': {'Start': 113.80498520498473,
'End': 118.82898520498473,
'Text': 'HE WANTS ME TO STAND READY TO OBEY ANY SUMMONS THE POLICE MAY SEND ME'},
'Audio': array([0.00015562, 0.00071042, 0.00115697, ..., 0. , 0. ,
0. ])},
'Speaker 4': {'Segment 1': {'Start': 3.4360869201656032,
'End': 7.532086920165604,
'Text': 'OUR SCANDALOUS PROCEEDINGS OFTEN EXPOSED US TO THE GREATEST DANGER'},
'Segment 2': {'Start': 21.16833906298162,
'End': 24.52833906298162,
'Text': 'THIS IS THE AMUSING ADVENTURE WHICH CLOSED OUR EXPLOITS'},
'Segment 3': {'Start': 75.85712248174686,
'End': 78.48112248174685,
'Text': 'WE TOOK OUR THREE PRISONERS TO A LARGE BOAT'},
'Segment 4': {'Start': 90.22708634372017,
'End': 92.46708634372017,
'Text': 'WHERE IS MY HUSBAND'},
'Segment 5': {'Start': 93.89746801949082,
'End': 97.41746801949081,
'Text': 'NEVER FEAR YOU SHALL SEE HIM AGAIN TO MORROW'},
'Segment 6': {'Start': 99.22455077059192,
'End': 103.25655077059191,
'Text': 'TWO DAYS AFTERWARDS OUR NOCTURNAL ORGY BEGAN TO BE TALKED OF'},
'Audio': array([0., 0., 0., ..., 0., 0., 0.])}}
The organization of the provided label is as follows. The label is a dictionary containing each of the 4 speaker's active period (with Start and End denoting the starting and ending times in second) and the transcription for each segment. The Audio contains a 2-minute audio containing only that speaker's sound.
# label organization
print(label['Speaker 1']['Segment 2']['Start'])
print(label['Speaker 1']['Segment 2']['End'])
print(label['Speaker 1']['Segment 2']['Text'])
60.43879936299203 63.76679936299203 THE VERY EMPERORS HAD ARRAYED THEMSELVES ON HER SIDE
# listen to the speaker's 2-minute audio
Audio(label['Speaker 1']['Audio'], rate=sr)
With the help of the pyannote-metrics package, we can have a better visualization on the speakers' activations. Install this package by your own, and check its tutorial page for the proper usage.
# better visualization
import pickle
from pyannote.core import Segment, Timeline, Annotation
from pyannote.metrics.diarization import DiarizationErrorRate
# I've already saved the label, here we simply load it
reference_pyannote = pickle.load(open("diarization_label.pkl", "rb"))
# visualization
reference_pyannote
# or get the information
print(reference_pyannote)
[ 00:00:00.000 --> 00:00:02.688] _ Speaker 3 [ 00:00:03.436 --> 00:00:07.532] _ Speaker 4 [ 00:00:08.508 --> 00:00:11.612] _ Speaker 2 [ 00:00:09.766 --> 00:00:13.862] _ Speaker 3 [ 00:00:14.968 --> 00:00:18.520] _ Speaker 3 [ 00:00:18.958 --> 00:00:20.878] _ Speaker 3 [ 00:00:21.168 --> 00:00:24.528] _ Speaker 4 [ 00:00:22.723 --> 00:00:26.819] _ Speaker 1 [ 00:00:27.632 --> 00:00:31.280] _ Speaker 3 [ 00:00:29.542 --> 00:00:34.054] _ Speaker 2 [ 00:00:35.201 --> 00:00:40.417] _ Speaker 3 [ 00:00:41.471 --> 00:00:43.935] _ Speaker 2 [ 00:00:44.491 --> 00:00:46.187] _ Speaker 3 [ 00:00:46.490 --> 00:00:48.826] _ Speaker 2 [ 00:00:48.132 --> 00:00:51.588] _ Speaker 3 [ 00:00:52.042 --> 00:00:56.938] _ Speaker 3 [ 00:00:58.190 --> 00:01:03.246] _ Speaker 3 [ 00:01:00.438 --> 00:01:03.766] _ Speaker 1 [ 00:01:04.512 --> 00:01:07.968] _ Speaker 3 [ 00:01:08.576 --> 00:01:13.760] _ Speaker 2 [ 00:01:12.462 --> 00:01:15.118] _ Speaker 3 [ 00:01:15.857 --> 00:01:18.481] _ Speaker 4 [ 00:01:19.112 --> 00:01:22.952] _ Speaker 3 [ 00:01:21.566 --> 00:01:25.438] _ Speaker 1 [ 00:01:26.057 --> 00:01:29.737] _ Speaker 1 [ 00:01:30.227 --> 00:01:32.467] _ Speaker 4 [ 00:01:32.739 --> 00:01:36.643] _ Speaker 1 [ 00:01:33.897 --> 00:01:37.417] _ Speaker 4 [ 00:01:38.420 --> 00:01:39.892] _ Speaker 3 [ 00:01:39.224 --> 00:01:43.256] _ Speaker 4 [ 00:01:43.945 --> 00:01:47.657] _ Speaker 3 [ 00:01:48.655 --> 00:01:53.007] _ Speaker 2 [ 00:01:53.804 --> 00:01:58.828] _ Speaker 3
Your meeting transcription system should do the following two things:
Here is an example about the design of the pipeline:
After this step, each segment should contain 1 or 2 single-speaker utterances. In the cases where the utterances are not fully overlapped in the segment, recalculate the start/end times of all the utterances. 3. For each single-speaker utterance in each segment, train a speaker identification module to determine the speaker information. 4. For each single-speaker utterance in each segment, train an ASR module to transcribe it into text.
You can also use any other pipelines that you come up with, only to make sure that your system's output should be both the speaker activity annotation like the one above, and the transcribed texts for each of the speakers. You can use any of the released sample solultions for the 4 coding homework in this project. Again, remember that you only need to *overfit* your model on this 2-minute sample.
We will not have a strict performance requirement for this final project and the grading will be based on your overall implementation. Try your best to achieve a good enough accuracy!
# TODO: build your meeting transcription system
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
from itertools import permutations
# visualize the audio
plt.plot(audio)
ticks_in_second = np.arange(len(audio)) / sr
max_second = np.floor(ticks_in_second[-1])
plt.xticks(ticks=np.arange(0, (max_second+1)*sr, (max_second+1)*sr/4),
labels=np.arange(0, (max_second+1), (max_second+1)/4))
plt.xlabel('Time (s)')
plt.tight_layout()
# calculate Mel Spectrogram using 64 filters for Mel filterbank
# audio spectrogram magnitude: (257, 7501)
audio_spec = librosa.stft(audio, n_fft=512, hop_length=256)
audio_spec_mag = np.abs(audio_spec)
# visualize part of it
plt.figure(figsize=(10, 5))
plt.imshow(audio_spec_mag[:,1500:2000]**0.33, origin='lower')
plt.tight_layout()
# Create labels for VAD
labels = np.any(audio_spec_mag>0,axis=0).astype(int)
# generate data for train, validation, the test dataset is the whole dataset
train_data = audio_spec_mag
val_data = audio_spec_mag[:,750:1500]
test_data = audio_spec_mag[:,:750]
train_label = labels
val_label = labels[750:1500]
test_label = labels[:750]
# MVN
train_mean = np.mean(train_data, 1)
train_var = np.var(train_data, 1)
train_data = (train_data - train_mean[:,np.newaxis]) / train_var[:,np.newaxis]
val_data = (val_data - train_mean[:,np.newaxis]) / train_var[:,np.newaxis]
test_data = (test_data - train_mean[:,np.newaxis]) / train_var[:,np.newaxis]
# the function to calcualte the accuracy
def Accuracy(predicts, labels):
'''
Compute accuracy of predicted labels against true labels.
args:
predicts: binary array or tensor of shape (num_frame, )
labels: binary array or tensor of shape (num_frame, )
return:
scalar float accuracy between 0 and 1
'''
assert len(predicts) == len(labels)
result = (predicts == labels)
if torch.is_tensor(result):
return (sum(result)/len(result)).to(torch.float32).item()
return sum(result)/len(result)
# visualize the labels
plt.figure(figsize=(20, 5))
plt.plot(labels)
plt.tight_layout()
from sklearn.svm import LinearSVC
# Instantiate a SVM classifier
svm = LinearSVC(random_state=0, max_iter=1000)
# Train your classifier
svm.fit(train_data.transpose(1,0),train_label)
# Make prediction on the training, validation, and test sets
train_predict_svm = svm.predict(train_data.transpose(1,0))
val_predict_svm = svm.predict(val_data.transpose(1,0))
test_predict_svm = svm.predict(test_data.transpose(1,0))
# # Calculate accuracy on the training, validation, and test sets
train_acc_svm = Accuracy(train_predict_svm, train_label)
val_acc_svm = Accuracy(val_predict_svm, val_label)
test_acc_svm = Accuracy(test_predict_svm, test_label)
print(f'Train set accuracy: {str(round(train_acc_svm*100, 1))}%')
print(f'Val set accuracy: {str(round(val_acc_svm*100, 1))}%')
print(f'Test set accuracy: {str(round(test_acc_svm*100, 1))}%')
Train set accuracy: 95.9% Val set accuracy: 98.9% Test set accuracy: 96.0%
e:\Python\lib\site-packages\sklearn\svm\_base.py:1244: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations. warnings.warn(
# spoken_status, 1 speech segment, 0 nonspeech segement
spoken_status = svm.predict(train_data.transpose(1,0))
# visualize the spoken_status
plt.figure(figsize=(20, 5))
plt.plot(spoken_status)
plt.tight_layout()
# define the smoothing function
# increase accuracy, and make sure the small number of 0 between 1 is diminished
def smoothing(prediction, context=47):
# prediction: shape (num_frame,)
smooth_pred = np.convolve(prediction, np.ones(context)/context, 'valid')
pred = np.pad(smooth_pred, (context//2, context//2), 'edge')
return pred
smooth_spoken_status = smoothing(spoken_status).round()
print(f"After smoothing, prediction Accuracy is {Accuracy(smooth_spoken_status, labels)}")
After smoothing, prediction Accuracy is 0.9644047460338622
def check_nonzero_arrays(arr1, arr2, arr3, arr4):
non_zero_arrays = {}
if np.count_nonzero(arr1) > 0:
if np.count_nonzero(arr1) > 30:
non_zero_arrays[0] = (np.count_nonzero(arr1))
if np.count_nonzero(arr2) > 0:
if np.count_nonzero(arr2) > 30:
non_zero_arrays[1] = (np.count_nonzero(arr2))
if np.count_nonzero(arr3) > 0:
if np.count_nonzero(arr3) > 30:
non_zero_arrays[2] = (np.count_nonzero(arr3))
if np.count_nonzero(arr4) > 0:
if np.count_nonzero(arr4) > 30:
non_zero_arrays[3] = (np.count_nonzero(arr4))
if len(non_zero_arrays) >= 2:
if len(non_zero_arrays) > 2:
# only keep the top 2 speakers
sorted_values = sorted(non_zero_arrays.values(), reverse=True)
top_two_values = sorted_values[:2]
non_zero_arrays = {key: value for key, value in non_zero_arrays.items() if value in top_two_values}
return 1, list(non_zero_arrays.keys())
else:
return 0, list(non_zero_arrays.keys())
# generate labels
# for speech, nonspeech area, if the speech area less than 25 length, make it 0, unspeech.
# record the start and end index of each grouping segment
# for each segment, create label by find if at least two spk speak at this area.
def overlap_grouping(mix_mag, pred, label, threshold=30):
# starts is the index for start of each 1, [0,1,1,0], starts = [1]
starts, = np.where(np.diff(pred, prepend=0) == 1)
# end is the index for end of each 1, starts = [2]
# starts:end+1 is the slice
ends, = np.where(np.diff(pred, append=0) == -1)
# l is the length of the segmentation
l = ends-starts+1
start_list = []
end_list = []
len_list = []
# if the length is less than 25, delete this group
for group in range(len(l)):
if l[group] > threshold:
start_list.append(starts[group])
end_list.append(ends[group])
len_list.append(l[group])
start_list, end_list, len_list = np.array(start_list), np.array(end_list), np.array(len_list)
# Get the clean spk audio
spk1_audio = label['Speaker 1']['Audio']
spk2_audio = label['Speaker 2']['Audio']
spk3_audio = label['Speaker 3']['Audio']
spk4_audio = label['Speaker 4']['Audio']
# get the mag of them
spk1_mag = np.abs(librosa.stft(spk1_audio, n_fft=512, hop_length=256))
spk2_mag = np.abs(librosa.stft(spk2_audio, n_fft=512, hop_length=256))
spk3_mag = np.abs(librosa.stft(spk3_audio, n_fft=512, hop_length=256))
spk4_mag = np.abs(librosa.stft(spk4_audio, n_fft=512, hop_length=256))
# for each group, if any two of them overlap, then the labels is 1
overlap_label = np.zeros_like(start_list)
# find the largest length
max_len = max(len_list)
# the mixture data for each group
data = np.empty((mix_mag.shape[0]*max_len,0))
# the spk for each group, two speakers
spks = []
mix_spec = {}
separated_spec = {}
all_spec = {}
all_spk = []
# for each group
for i in range(len(overlap_label)):
# get the index of the mag
starts_index = start_list[i]
end_index = end_list[i] + 1
# the mix mag for this group, (257, length)
mix = mix_mag[:,starts_index:end_index]
# pad the mag to the max, (257*max,1)
padded_array = np.pad(mix, ((0, 0), (0, max_len-len_list[i])), mode='constant')
data = np.concatenate((data, padded_array.reshape(-1,1)), axis=1)
# get the magnitude of this group
spk1 = spk1_mag[:,starts_index:end_index].flatten()
spk2 = spk2_mag[:,starts_index:end_index].flatten()
spk3 = spk3_mag[:,starts_index:end_index].flatten()
spk4 = spk4_mag[:,starts_index:end_index].flatten()
spk_sepc_mag = [spk1_mag[:,starts_index:end_index], spk2_mag[:,starts_index:end_index], spk3_mag[:,starts_index:end_index], spk4_mag[:,starts_index:end_index]]
all_spec[i] = padded_array
overlap_label[i], spk = check_nonzero_arrays(spk1, spk2, spk3, spk4)
all_spk.append(spk)
if overlap_label[i] > 0:
# pad the two seperated spec to the max_len, (257,max_len,1)
s = np.empty((mix_mag.shape[0],max_len,0))
s = np.concatenate((s,np.pad(spk_sepc_mag[spk[0]], ((0, 0), (0, max_len-len_list[i])), mode='constant')[:,:,np.newaxis]), axis=2)
s = np.concatenate((s,np.pad(spk_sepc_mag[spk[1]], ((0, 0), (0, max_len-len_list[i])), mode='constant')[:,:,np.newaxis]), axis=2)
# get the seperated data for two spks
separated_spec[i] = s
# get the mixture data
mix_spec[i] = padded_array
spks.append(spk)
return overlap_label, data, mix_spec, separated_spec, spks, start_list, end_list, all_spec, all_spk
# get the speech area, and the overlap label for each group(speech area)
overlap_label, segmentation_mix_mag, mix_spec_dic, seperated_spec_dict, spk_list, starts, ends, all_mix_spec, all_spk = overlap_grouping(audio_spec_mag, smooth_spoken_status, label)
train_mean = np.mean(segmentation_mix_mag, 1)
train_var = np.var(segmentation_mix_mag, 1)
segmentation_data = (segmentation_mix_mag - train_mean[:,np.newaxis]) / train_var[:,np.newaxis]
from sklearn.svm import LinearSVC
# Instantiate a SVM classifier
svm_overlap = LinearSVC(random_state=0, max_iter=1000)
# Train your classifier
svm_overlap.fit(segmentation_data.transpose(1,0),overlap_label)
# Make prediction on the training, validation, and test sets
train_predict_svm = svm_overlap.predict(segmentation_data.transpose(1,0))
# # Calculate accuracy on the training, validation, and test sets
train_acc_svm = Accuracy(train_predict_svm, overlap_label)
print(f'Train set accuracy: {str(round(train_acc_svm*100, 1))}%')
Train set accuracy: 100.0%
e:\Python\lib\site-packages\sklearn\svm\_base.py:1244: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations. warnings.warn(
# Now we have the overlap status, we need which segmentation is overlap
overlap_status = train_predict_svm
overlap_status
array([0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0,
0, 0], dtype=int64)
# 1. get the index of the overlap_status as new list overlap: get which segmentation has the overlap status
# 2. create dataloader, sample index i, overlap[i] = 2, segmentation 2,
# mixture_input = mix_spec_dic[overlap[i]], (257, 492)
# the label: sperated_spec = seperated_spec_dict[overlap[i]], (257, 492, 2)
# create data loader pipeline
from torch.utils.data import Dataset, DataLoader
batch_size = 2
class dataset_pipeline(Dataset):
def __init__(self, mix_spec_dic, seperated_spec_dict, overlap_label):
super(dataset_pipeline, self).__init__()
self.mix_mag_dic = mix_spec_dic
self.sep_mag_dic = seperated_spec_dict
self.seg_overlap = overlap_label
self.overlap = np.where(overlap_label==1)[0]
# the number of overlap segmentations
self._len = len(self.overlap)
def __getitem__(self, index):
# (257, 492)
mixture_mag_spec = torch.from_numpy(self.mix_mag_dic[self.overlap[index]])
# sperated_spec (257, 492, 2)
seperated_spec = seperated_spec_dict[self.overlap[index]]
# speech1_mag_spec (257, 492)
speech1_mag_spec = torch.from_numpy(seperated_spec[:,:,0])
# speech2_mag_spec (257, 492)
speech2_mag_spec = torch.from_numpy(seperated_spec[:,:,1])
# target WFMs, each is (freq, time) = (257, 492)
eps = 1e-6 # numerical stability
spk1_WFM = speech1_mag_spec**2 / (speech1_mag_spec**2 + speech2_mag_spec**2 + eps)
spk2_WFM = speech2_mag_spec**2 / (speech1_mag_spec**2 + speech2_mag_spec**2 + eps)
# get seperated spec (num_target, freq, time)
separated_spk1_spec = spk1_WFM * mixture_mag_spec
separated_spk2_spec = spk2_WFM * mixture_mag_spec
separated_spec = torch.stack([separated_spk1_spec, separated_spk2_spec])
return mixture_mag_spec, separated_spec
def __len__(self):
return self._len
# define data loaders
train_loader = DataLoader(dataset_pipeline(mix_spec_dic, seperated_spec_dict, overlap_label),
batch_size=batch_size,
shuffle=True,
)
validation_loader = DataLoader(dataset_pipeline(mix_spec_dic, seperated_spec_dict, overlap_label),
batch_size=1,
shuffle=False,
)
dataset_len = len(train_loader)
log_step = dataset_len // 4
# a 2-layer LSTM
class DeepLSTM(nn.Module):
def __init__(self, feature_dim):
super(DeepLSTM, self).__init__()
# layers, and activation function
self.feature_dim = feature_dim
self.hidden_unit = 128 # number of hidden units
# LSTM
self.layer1 = nn.LSTM(self.feature_dim, self.hidden_unit, batch_first = True, num_layers = 1, bidirectional = True)
self.layer2 = nn.LSTM(self.hidden_unit*2, self.hidden_unit, batch_first = True, num_layers = 1, bidirectional = True)
# output layer
# this is an FC layer
self.output = nn.Linear(self.hidden_unit*2, self.feature_dim)
self.Sigmoid = nn.Sigmoid()
# the function for the forward pass of network (i.e. from input to output)
def forward(self, input):
# the input is a batch of spectrograms with shape (batch_size, frequency_dim, time_step)
batch_size, freq_dim, time_step = input.shape
# note that LSTM layers require input with shape (batch_size, sequence_length, feature_dim)
# so here we need to reshape the input
# make it (batch, time, freq), swap the two dimensions
input = input.transpose(1, 2).contiguous()
# apply MVN
mean = torch.mean(input, dim=(0, 1), keepdim=True)
var = torch.var(input, dim=(0, 1), keepdim=True)
input = (input - mean) / torch.sqrt(var)
# pass through LSTM
output,_ = self.layer1(input)
output,_ = self.layer2(output)
# the output should have shape (batch, time, hidden_unit)
# pass to the output layer
# input for this FC layer (batch*time, freq)
output = self.output(output.contiguous().view(batch_size*time_step, -1))
# the output: (batch, time, freq)
output = self.Sigmoid(output)
# reshape back
output = output.view(batch_size, time_step, freq_dim) # (batch, time, freq)
# create another WFM
output2 = 1 - output
# stack them together and reshape
output = torch.stack([output, output2]) # (2, batch, time, freq)
output = output.permute(1, 0, 3, 2).contiguous() # (batch_size, 2, freq, time)
return output
model_LSTM = DeepLSTM(feature_dim=257)
optimizer = optim.Adam(model_LSTM.parameters(), lr=5e-3)
# the batch-level MSE objective with PIT
def PIT_MSE(estimation, target):
"""
args:
estimation: Pytorch tensor, shape (batch_size, num_target, freq, time)
target: Pytorch tensor, shape (batch_size, num_target, freq, time)
output:
mse: single scalar, average MSE across all dimensions
"""
# get the number of target
num_target = estimation.shape[1]
# get all the permutations
perms = permutations(range(num_target))
# assume MSE = infinite
mse = float('inf')
# calculate MSE for each perm and choose the smallest MSE
for perm in perms:
estimation_perm = estimation[:, perm, :, :]
mse_perm = torch.mean(torch.pow(estimation_perm - target, 2))
if mse_perm < mse:
mse = mse_perm
return mse
def train(model, epoch, versatile=True):
start_time = time.time()
model = model.train() # set the model to training mode. Always do this before you start training!
train_loss = 0.
# load batch data
for batch_idx, data in enumerate(train_loader):
# (Batch, feature, time) = (4,257,492)
mixture_mag_spec = data[0]
# (Batch, 2, feature, time) = (4,2,257,492)
target = data[1]
# clean up the gradients in the optimizer
# this should be called for each batch
optimizer.zero_grad()
# output: two WFMs (batch_size, 2, freq, time) = [4, 2, 257, 492]
output = model(mixture_mag_spec)
# get each WFM, [4, 257, 492]
spk1_WFM = output[:, 0, :, :]
spk2_WFM = output[:, 1, :, :]
# get seperated spec (Batch, freq, time) = [4, 257, 492]
separated_spk1_spec = spk1_WFM * mixture_mag_spec
separated_spk2_spec = spk2_WFM * mixture_mag_spec
# (num_target, batch, freq, time) = [2, 4, 257, 492]
estimation = torch.stack([separated_spk1_spec, separated_spk2_spec])
estimation = estimation.transpose(0, 1).contiguous() # (batch, num_target, freq, time), swap the two dimension back
# PIT_MSE as objective
loss = PIT_MSE(estimation, target)
# automatically calculate the backward pass
loss.backward()
# perform the actual backpropagation
optimizer.step()
train_loss += loss.data.item()
# print the training progress
if versatile:
if (batch_idx+1) % log_step == 0:
elapsed = time.time() - start_time
print('| epoch {:3d} | {:5d}/{:5d} batches | ms/batch {:5.2f} | MSE {:5.4f} |'.format(
epoch, batch_idx+1, len(train_loader),
elapsed * 1000 / (batch_idx+1),
train_loss / (batch_idx+1)
))
train_loss /= (batch_idx+1)
print('-' * 99)
print(' | end of training epoch {:3d} | time: {:5.2f}s | MSE {:5.4f} |'.format(
epoch, (time.time() - start_time), train_loss))
return train_loss
def validate(model, epoch):
start_time = time.time()
model = model.eval() # set the model to evaluation mode. Always do this during validation or test phase!
validation_loss = 0.
# load batch data
for batch_idx, data in enumerate(validation_loader):
mixture_mag_spec = data[0]
target = data[1]
# you don't need to calculate the backward pass and the gradients during validation
# so you can call torch.no_grad() to only calculate the forward pass to save time and memory
with torch.no_grad():
# output: two WFMs (batch_size, 2, freq, time)
output = model(mixture_mag_spec)
# get each WFM
spk1_WFM = output[:, 0, :, :]
spk2_WFM = output[:, 1, :, :]
# get seperated spec (num_target, freq, time)
separated_spk1_spec = spk1_WFM * mixture_mag_spec
separated_spk2_spec = spk2_WFM * mixture_mag_spec
estimation = torch.stack([separated_spk1_spec, separated_spk2_spec])
estimation = estimation.transpose(0, 1).contiguous() # (batch, num_target, freq, time), swap the two dimension back
# PIT_MSE as objective
loss = PIT_MSE(estimation, target)
validation_loss += loss.data.item()
validation_loss /= (batch_idx+1)
print(' | end of validation epoch {:3d} | time: {:5.2f}s | MSE {:5.4f} |'.format(
epoch, (time.time() - start_time), validation_loss))
print('-' * 99)
return validation_loss
# hyperparameters and model training
total_epoch = 150 # train the model for 150 epochs
model_save = 'best_model_Speech_separation.pt' # path to save the best validation model
# main function
training_loss = []
validation_loss = []
for epoch in range(1, total_epoch + 1):
training_loss.append(train(model_LSTM, epoch, versatile=False))
validation_loss.append(validate(model_LSTM, epoch))
if training_loss[-1] == np.min(training_loss):
print(' Best training model found.')
if validation_loss[-1] == np.min(validation_loss):
# save current best model on validation set
with open(model_save, 'wb') as f:
torch.save(model_LSTM.state_dict(), f)
print(' Best validation model found and saved.')
print('-' * 99)
---------------------------------------------------------------------------------------------------
| end of training epoch 1 | time: 1.71s | MSE 0.1765 |
| end of validation epoch 1 | time: 0.56s | MSE 0.1381 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 2 | time: 1.36s | MSE 0.1471 |
| end of validation epoch 2 | time: 0.60s | MSE 0.0998 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 3 | time: 1.43s | MSE 0.1135 |
| end of validation epoch 3 | time: 0.57s | MSE 0.0758 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 4 | time: 1.38s | MSE 0.0783 |
| end of validation epoch 4 | time: 0.56s | MSE 0.0730 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 5 | time: 1.36s | MSE 0.0635 |
| end of validation epoch 5 | time: 0.56s | MSE 0.0598 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 6 | time: 1.37s | MSE 0.0706 |
| end of validation epoch 6 | time: 0.60s | MSE 0.0551 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 7 | time: 1.38s | MSE 0.0541 |
| end of validation epoch 7 | time: 0.55s | MSE 0.0464 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 8 | time: 1.36s | MSE 0.0420 |
| end of validation epoch 8 | time: 0.56s | MSE 0.0370 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 9 | time: 1.36s | MSE 0.0417 |
| end of validation epoch 9 | time: 0.58s | MSE 0.0316 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 10 | time: 1.36s | MSE 0.0327 |
| end of validation epoch 10 | time: 0.56s | MSE 0.0345 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 11 | time: 1.34s | MSE 0.0319 |
| end of validation epoch 11 | time: 0.56s | MSE 0.0327 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 12 | time: 1.34s | MSE 0.0313 |
| end of validation epoch 12 | time: 0.56s | MSE 0.0276 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 13 | time: 1.35s | MSE 0.0265 |
| end of validation epoch 13 | time: 0.57s | MSE 0.0251 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 14 | time: 1.38s | MSE 0.0218 |
| end of validation epoch 14 | time: 0.56s | MSE 0.0242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 15 | time: 1.39s | MSE 0.0231 |
| end of validation epoch 15 | time: 0.57s | MSE 0.0225 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 16 | time: 1.39s | MSE 0.0201 |
| end of validation epoch 16 | time: 0.58s | MSE 0.0216 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 17 | time: 1.39s | MSE 0.0192 |
| end of validation epoch 17 | time: 0.58s | MSE 0.0203 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 18 | time: 1.37s | MSE 0.0181 |
| end of validation epoch 18 | time: 0.57s | MSE 0.0203 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 19 | time: 1.38s | MSE 0.0185 |
| end of validation epoch 19 | time: 0.56s | MSE 0.0190 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 20 | time: 1.38s | MSE 0.0287 |
| end of validation epoch 20 | time: 0.58s | MSE 0.0180 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 21 | time: 1.38s | MSE 0.0166 |
| end of validation epoch 21 | time: 0.58s | MSE 0.0192 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 22 | time: 1.41s | MSE 0.0169 |
| end of validation epoch 22 | time: 0.59s | MSE 0.0176 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 23 | time: 1.37s | MSE 0.0157 |
| end of validation epoch 23 | time: 0.57s | MSE 0.0169 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 24 | time: 1.38s | MSE 0.0152 |
| end of validation epoch 24 | time: 0.59s | MSE 0.0162 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 25 | time: 1.39s | MSE 0.0150 |
| end of validation epoch 25 | time: 0.57s | MSE 0.0161 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 26 | time: 1.38s | MSE 0.0144 |
| end of validation epoch 26 | time: 0.57s | MSE 0.0158 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 27 | time: 1.38s | MSE 0.0137 |
| end of validation epoch 27 | time: 0.56s | MSE 0.0153 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 28 | time: 1.37s | MSE 0.0147 |
| end of validation epoch 28 | time: 0.57s | MSE 0.0147 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 29 | time: 1.42s | MSE 0.0136 |
| end of validation epoch 29 | time: 0.59s | MSE 0.0147 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 30 | time: 1.42s | MSE 0.0141 |
| end of validation epoch 30 | time: 0.57s | MSE 0.0145 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 31 | time: 1.38s | MSE 0.0134 |
| end of validation epoch 31 | time: 0.57s | MSE 0.0142 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 32 | time: 1.38s | MSE 0.0133 |
| end of validation epoch 32 | time: 0.60s | MSE 0.0142 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 33 | time: 1.42s | MSE 0.0132 |
| end of validation epoch 33 | time: 0.58s | MSE 0.0140 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 34 | time: 1.38s | MSE 0.0127 |
| end of validation epoch 34 | time: 0.58s | MSE 0.0137 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 35 | time: 1.39s | MSE 0.0129 |
| end of validation epoch 35 | time: 0.57s | MSE 0.0136 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 36 | time: 1.39s | MSE 0.0125 |
| end of validation epoch 36 | time: 0.57s | MSE 0.0134 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 37 | time: 1.37s | MSE 0.0221 |
| end of validation epoch 37 | time: 0.57s | MSE 0.0133 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 38 | time: 1.34s | MSE 0.0123 |
| end of validation epoch 38 | time: 0.56s | MSE 0.0132 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 39 | time: 1.34s | MSE 0.0121 |
| end of validation epoch 39 | time: 0.58s | MSE 0.0131 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 40 | time: 1.35s | MSE 0.0122 |
| end of validation epoch 40 | time: 0.56s | MSE 0.0130 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 41 | time: 1.33s | MSE 0.0120 |
| end of validation epoch 41 | time: 0.56s | MSE 0.0129 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 42 | time: 1.37s | MSE 0.0117 |
| end of validation epoch 42 | time: 0.57s | MSE 0.0128 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 43 | time: 1.39s | MSE 0.0117 |
| end of validation epoch 43 | time: 0.58s | MSE 0.0128 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 44 | time: 1.41s | MSE 0.0214 |
| end of validation epoch 44 | time: 0.58s | MSE 0.0127 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 45 | time: 1.36s | MSE 0.0117 |
| end of validation epoch 45 | time: 0.57s | MSE 0.0127 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 46 | time: 1.36s | MSE 0.0117 |
| end of validation epoch 46 | time: 0.56s | MSE 0.0126 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 47 | time: 1.35s | MSE 0.0116 |
| end of validation epoch 47 | time: 0.57s | MSE 0.0125 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 48 | time: 1.39s | MSE 0.0211 |
| end of validation epoch 48 | time: 0.57s | MSE 0.0125 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 49 | time: 1.40s | MSE 0.0114 |
| end of validation epoch 49 | time: 0.56s | MSE 0.0124 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 50 | time: 1.37s | MSE 0.0112 |
| end of validation epoch 50 | time: 0.58s | MSE 0.0124 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 51 | time: 1.36s | MSE 0.0115 |
| end of validation epoch 51 | time: 0.57s | MSE 0.0123 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 52 | time: 1.37s | MSE 0.0114 |
| end of validation epoch 52 | time: 0.58s | MSE 0.0123 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 53 | time: 1.41s | MSE 0.0113 |
| end of validation epoch 53 | time: 0.60s | MSE 0.0123 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 54 | time: 1.44s | MSE 0.0209 |
| end of validation epoch 54 | time: 0.58s | MSE 0.0122 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 55 | time: 1.41s | MSE 0.0112 |
| end of validation epoch 55 | time: 0.58s | MSE 0.0122 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 56 | time: 1.38s | MSE 0.0208 |
| end of validation epoch 56 | time: 0.58s | MSE 0.0122 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 57 | time: 1.38s | MSE 0.0208 |
| end of validation epoch 57 | time: 0.58s | MSE 0.0122 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 58 | time: 1.39s | MSE 0.0110 |
| end of validation epoch 58 | time: 0.58s | MSE 0.0121 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 59 | time: 1.37s | MSE 0.0111 |
| end of validation epoch 59 | time: 0.57s | MSE 0.0121 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 60 | time: 1.38s | MSE 0.0111 |
| end of validation epoch 60 | time: 0.56s | MSE 0.0121 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 61 | time: 1.38s | MSE 0.0110 |
| end of validation epoch 61 | time: 0.57s | MSE 0.0121 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 62 | time: 1.38s | MSE 0.0110 |
| end of validation epoch 62 | time: 0.56s | MSE 0.0120 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 63 | time: 1.38s | MSE 0.0112 |
| end of validation epoch 63 | time: 0.56s | MSE 0.0120 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 64 | time: 1.35s | MSE 0.0109 |
| end of validation epoch 64 | time: 0.55s | MSE 0.0120 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 65 | time: 1.32s | MSE 0.0109 |
| end of validation epoch 65 | time: 0.56s | MSE 0.0119 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 66 | time: 1.33s | MSE 0.0205 |
| end of validation epoch 66 | time: 0.56s | MSE 0.0119 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 67 | time: 1.36s | MSE 0.0108 |
| end of validation epoch 67 | time: 0.54s | MSE 0.0119 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 68 | time: 1.32s | MSE 0.0108 |
| end of validation epoch 68 | time: 0.56s | MSE 0.0119 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 69 | time: 1.33s | MSE 0.0109 |
| end of validation epoch 69 | time: 0.56s | MSE 0.0119 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 70 | time: 1.39s | MSE 0.0204 |
| end of validation epoch 70 | time: 0.57s | MSE 0.0118 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 71 | time: 1.35s | MSE 0.0204 |
| end of validation epoch 71 | time: 0.56s | MSE 0.0118 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 72 | time: 1.38s | MSE 0.0107 |
| end of validation epoch 72 | time: 0.57s | MSE 0.0118 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 73 | time: 1.39s | MSE 0.0107 |
| end of validation epoch 73 | time: 0.57s | MSE 0.0118 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 74 | time: 1.36s | MSE 0.0108 |
| end of validation epoch 74 | time: 0.58s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 75 | time: 1.35s | MSE 0.0106 |
| end of validation epoch 75 | time: 0.56s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 76 | time: 1.34s | MSE 0.0107 |
| end of validation epoch 76 | time: 0.56s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 77 | time: 1.35s | MSE 0.0106 |
| end of validation epoch 77 | time: 0.56s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 78 | time: 1.34s | MSE 0.0106 |
| end of validation epoch 78 | time: 0.56s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 79 | time: 1.34s | MSE 0.0107 |
| end of validation epoch 79 | time: 0.55s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 80 | time: 1.33s | MSE 0.0107 |
| end of validation epoch 80 | time: 0.55s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 81 | time: 1.33s | MSE 0.0107 |
| end of validation epoch 81 | time: 0.56s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 82 | time: 1.33s | MSE 0.0107 |
| end of validation epoch 82 | time: 0.55s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 83 | time: 1.34s | MSE 0.0106 |
| end of validation epoch 83 | time: 0.55s | MSE 0.0117 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 84 | time: 1.34s | MSE 0.0107 |
| end of validation epoch 84 | time: 0.56s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 85 | time: 1.34s | MSE 0.0107 |
| end of validation epoch 85 | time: 0.56s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 86 | time: 1.34s | MSE 0.0106 |
| end of validation epoch 86 | time: 0.56s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 87 | time: 1.34s | MSE 0.0202 |
| end of validation epoch 87 | time: 0.56s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 88 | time: 1.34s | MSE 0.0105 |
| end of validation epoch 88 | time: 0.56s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 89 | time: 1.33s | MSE 0.0105 |
| end of validation epoch 89 | time: 0.56s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 90 | time: 1.33s | MSE 0.0105 |
| end of validation epoch 90 | time: 0.56s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 91 | time: 1.40s | MSE 0.0106 |
| end of validation epoch 91 | time: 0.58s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 92 | time: 1.39s | MSE 0.0106 |
| end of validation epoch 92 | time: 0.56s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 93 | time: 1.40s | MSE 0.0105 |
| end of validation epoch 93 | time: 0.56s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 94 | time: 1.38s | MSE 0.0105 |
| end of validation epoch 94 | time: 0.59s | MSE 0.0116 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 95 | time: 1.38s | MSE 0.0106 |
| end of validation epoch 95 | time: 0.60s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 96 | time: 1.43s | MSE 0.0105 |
| end of validation epoch 96 | time: 0.57s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 97 | time: 1.39s | MSE 0.0201 |
| end of validation epoch 97 | time: 0.57s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 98 | time: 1.35s | MSE 0.0104 |
| end of validation epoch 98 | time: 0.58s | MSE 0.0115 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 99 | time: 1.34s | MSE 0.0104 |
| end of validation epoch 99 | time: 0.56s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 100 | time: 1.34s | MSE 0.0200 |
| end of validation epoch 100 | time: 0.58s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 101 | time: 1.42s | MSE 0.0104 |
| end of validation epoch 101 | time: 0.59s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 102 | time: 1.40s | MSE 0.0104 |
| end of validation epoch 102 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 103 | time: 1.39s | MSE 0.0104 |
| end of validation epoch 103 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 104 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 104 | time: 0.56s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 105 | time: 1.34s | MSE 0.0103 |
| end of validation epoch 105 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 106 | time: 1.34s | MSE 0.0104 |
| end of validation epoch 106 | time: 0.56s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 107 | time: 1.39s | MSE 0.0200 |
| end of validation epoch 107 | time: 0.56s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 108 | time: 1.38s | MSE 0.0199 |
| end of validation epoch 108 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 109 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 109 | time: 0.55s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 110 | time: 1.35s | MSE 0.0103 |
| end of validation epoch 110 | time: 0.58s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 111 | time: 1.38s | MSE 0.0103 |
| end of validation epoch 111 | time: 0.58s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 112 | time: 1.39s | MSE 0.0104 |
| end of validation epoch 112 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 113 | time: 1.37s | MSE 0.0103 |
| end of validation epoch 113 | time: 0.57s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 114 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 114 | time: 0.58s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 115 | time: 1.40s | MSE 0.0104 |
| end of validation epoch 115 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 116 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 116 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 117 | time: 1.42s | MSE 0.0103 |
| end of validation epoch 117 | time: 0.59s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 118 | time: 1.41s | MSE 0.0102 |
| end of validation epoch 118 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 119 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 119 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 120 | time: 1.38s | MSE 0.0104 |
| end of validation epoch 120 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 121 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 121 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 122 | time: 1.35s | MSE 0.0103 |
| end of validation epoch 122 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 123 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 123 | time: 0.56s | MSE 0.0114 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 124 | time: 1.37s | MSE 0.0103 |
| end of validation epoch 124 | time: 0.55s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 125 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 125 | time: 0.58s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 126 | time: 1.38s | MSE 0.0102 |
| end of validation epoch 126 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 127 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 127 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 128 | time: 1.36s | MSE 0.0103 |
| end of validation epoch 128 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 129 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 129 | time: 0.55s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 130 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 130 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 131 | time: 1.37s | MSE 0.0102 |
| end of validation epoch 131 | time: 0.57s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 132 | time: 1.37s | MSE 0.0102 |
| end of validation epoch 132 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 133 | time: 1.37s | MSE 0.0198 |
| end of validation epoch 133 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 134 | time: 1.37s | MSE 0.0103 |
| end of validation epoch 134 | time: 0.58s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 135 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 135 | time: 0.55s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 136 | time: 1.35s | MSE 0.0102 |
| end of validation epoch 136 | time: 0.56s | MSE 0.0113 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 137 | time: 1.37s | MSE 0.0102 |
| end of validation epoch 137 | time: 0.56s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 138 | time: 1.38s | MSE 0.0101 |
| end of validation epoch 138 | time: 0.57s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 139 | time: 1.41s | MSE 0.0102 |
| end of validation epoch 139 | time: 0.58s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 140 | time: 1.42s | MSE 0.0101 |
| end of validation epoch 140 | time: 0.59s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 141 | time: 1.42s | MSE 0.0102 |
| end of validation epoch 141 | time: 0.58s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 142 | time: 1.40s | MSE 0.0102 |
| end of validation epoch 142 | time: 0.60s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 143 | time: 1.43s | MSE 0.0101 |
| end of validation epoch 143 | time: 0.60s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 144 | time: 1.42s | MSE 0.0102 |
| end of validation epoch 144 | time: 0.58s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 145 | time: 1.40s | MSE 0.0102 |
| end of validation epoch 145 | time: 0.59s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 146 | time: 1.40s | MSE 0.0101 |
| end of validation epoch 146 | time: 0.61s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 147 | time: 1.42s | MSE 0.0101 |
| end of validation epoch 147 | time: 0.60s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 148 | time: 1.43s | MSE 0.0198 |
| end of validation epoch 148 | time: 0.60s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 149 | time: 1.41s | MSE 0.0101 |
| end of validation epoch 149 | time: 0.60s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 150 | time: 1.41s | MSE 0.0101 |
| end of validation epoch 150 | time: 0.59s | MSE 0.0112 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
# plot the loss of the seperation model
plt.plot(training_loss, label='Training Loss')
plt.plot(validation_loss, label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
<matplotlib.legend.Legend at 0x19f004bf3a0>
model_save = 'best_model_Speech_separation.pt' # path to save the best validation model
model_LSTM.load_state_dict(torch.load(model_save))
validate(model_LSTM, 1)
| end of validation epoch 1 | time: 0.53s | MSE 0.0112 | ---------------------------------------------------------------------------------------------------
0.011175211863411942
model_LSTM = model_LSTM.eval() # set the model to evaluation mode. Always do this during validation or test phase!
mixture_mag_spec = mix_spec_dic[2]
target = seperated_spec_dict[2]
with torch.no_grad():
# output: two WFMs (batch_size, 2, freq, time)
output = model_LSTM(torch.from_numpy(mixture_mag_spec[np.newaxis,:,:]))
# get each WFM
spk1_WFM = output[:, 0, :, :]
spk2_WFM = output[:, 1, :, :]
# get seperated spec [1, 257, 492]
separated_spk1_spec = (spk1_WFM * mixture_mag_spec[np.newaxis,:,:]).squeeze()
separated_spk2_spec = (spk2_WFM * mixture_mag_spec[np.newaxis,:,:]).squeeze()
# this is the sepc that has phase
example_audio_spec = audio_spec[:, starts[2]:ends[2]+1]
mixture_mag_spec = mixture_mag_spec[:, :example_audio_spec.shape[1]]
separated_spk1_spec = separated_spk1_spec[:, :example_audio_spec.shape[1]]
separated_spk2_spec = separated_spk2_spec[:, :example_audio_spec.shape[1]]
# the true audio
phase = np.angle(example_audio_spec)
mix_stft_result = mixture_mag_spec * np.exp(1j * phase)
y = librosa.istft(np.asarray(mix_stft_result), win_length=512, hop_length=256)
Audio(y, rate=sr)
spk1_stft_result = separated_spk1_spec * np.exp(1j * phase)
y = librosa.istft(np.asarray(spk1_stft_result), win_length=512, hop_length=256)
Audio(y, rate=sr)
spk2_stft_result = separated_spk2_spec * np.exp(1j * phase)
y = librosa.istft(np.asarray(spk2_stft_result), win_length=512, hop_length=256)
Audio(y, rate=sr)
For this part, we have 24 segmentations.
For each segmentation i, if it is no overlap by known by overlap_status, then we get its mag from all_mix_spec[i] (257, 492), and spk from all_spk[i][0]: 2. Add magnitude to dict, Add spk to spk_label.
if it is overlap by known by overlap_label, we get its mix mag from all_mix_spec[i] (257, 492), Then we get its two seperated mag from the Seperation model, then get spk from all_spk[i]: [1,2]. Add magnitude to dict, Add spk to spk_label.
# convert the index from mag to the timeline
def index_time_transform(index, sr=16000):
original_index = index*256
t = original_index/sr
return t
def SIM_data(all_mix_spec, all_spk, overlap_status, model_LSTM, starts, ends):
magnitudes = []
spk_label = []
starts_time = []
ends_time = []
# for each segmentation
for i in range(len(all_mix_spec)):
# if it only have one spk
if overlap_status[i] == 0:
# add the magnitudes
magnitudes.append(all_mix_spec[i])
# add the spk label, 0~3
spk_label.append(all_spk[i][0])
# add the start time
starts_time.append(starts[i])
ends_time.append(ends[i])
# if it have two spks
else:
model_LSTM = model_LSTM.eval()
mixture_mag_spec = all_mix_spec[i]
with torch.no_grad():
# output: two WFMs (batch_size, 2, freq, time)
output = model_LSTM(torch.from_numpy(mixture_mag_spec[np.newaxis,:,:]))
# get each WFM
spk1_WFM = output[:, 0, :, :]
spk2_WFM = output[:, 1, :, :]
# get seperated spec [1, 257, 492]
separated_spk1_spec = (spk1_WFM * mixture_mag_spec[np.newaxis,:,:]).squeeze()
separated_spk2_spec = (spk2_WFM * mixture_mag_spec[np.newaxis,:,:]).squeeze()
# add the magnitudes 1
magnitudes.append(separated_spk1_spec)
# add the spk label, 0~3
spk_label.append(all_spk[i][0])
# add the start time
starts_time.append(starts[i])
ends_time.append(ends[i])
# add the magnitudes 1
magnitudes.append(separated_spk2_spec)
# add the spk label, 0~3
spk_label.append(all_spk[i][1])
# add the start time
starts_time.append(starts[i])
ends_time.append(ends[i])
return np.array(magnitudes), np.array(spk_label), np.array(starts_time), np.array(ends_time)
magnitudes, spk_label, starts_time, ends_time = SIM_data(all_mix_spec, all_spk, overlap_status, model_LSTM, starts, ends)
C:\Users\91688\AppData\Local\Temp\ipykernel_24056\3359080599.py:55: FutureWarning: The input object of type 'Tensor' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'Tensor', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`. return np.array(magnitudes), np.array(spk_label), np.array(starts_time), np.array(ends_time) C:\Users\91688\AppData\Local\Temp\ipykernel_24056\3359080599.py:55: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. return np.array(magnitudes), np.array(spk_label), np.array(starts_time), np.array(ends_time)
batch_size = 4
class SIM_data(Dataset):
def __init__(self, magnitudes, spk_label):
super(SIM_data, self).__init__()
self.magnitudes = magnitudes
self.spk_label = torch.from_numpy(spk_label).long()
self._len = len(magnitudes)
def __getitem__(self, index):
mag_spec = self.magnitudes[index]
if not torch.is_tensor(self.magnitudes[index]):
mag_spec = torch.from_numpy(self.magnitudes[index])
label = self.spk_label[index]
return mag_spec, label
def __len__(self):
return self._len
# define data loaders
train_loader = DataLoader(SIM_data(magnitudes, spk_label),
batch_size=batch_size,
shuffle=True, # this ensures that the sequential order of the training samples will be shuffled for different training epochs
)
validation_loader = DataLoader(SIM_data(magnitudes, spk_label),
batch_size=1,
shuffle=False, # typically we fix the sequential order of the validation samples
)
dataset_len = len(train_loader)
# borrow the implementation of AlexNet from https://pytorch.org/vision/stable/_modules/torchvision/models/alexnet.html#alexnet
class AlexNet(nn.Module):
def __init__(self, num_classes=4):
super(AlexNet, self).__init__()
self.bn = nn.BatchNorm2d(num_features=1)
self.features = nn.Sequential(
nn.Conv2d(1, 16, kernel_size=11, stride=4, padding=2), # number of input channel is 1 (for image it is 3)
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
nn.Conv2d(16, 32, kernel_size=5, padding=2), # we make the number of hidden channels smaller in these layers
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
nn.Conv2d(32, 32, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.Conv2d(32, 64, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.Conv2d(64, 64, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),
)
self.avgpool = nn.AdaptiveAvgPool2d((3, 3)) # perform adaptive mean pooling on any size of the input to match the provided size
self.classifier = nn.Sequential(
nn.Dropout(),
nn.Linear(64 * 3 * 3, 256),
nn.ReLU(inplace=True),
nn.Dropout(),
nn.Linear(256, 256),
nn.ReLU(inplace=True),
nn.Linear(256, num_classes),
)
def forward(self, x):
x = self.bn(x)
x = self.features(x)
x = self.avgpool(x) # the dimension after adaptive average pooling is (batch, 64, 3, 3)
x = torch.flatten(x, 1) # average
x = self.classifier(x)
return x
# test it with a sample input
model = AlexNet()
sample_input = torch.randn(2, 1, 257, 492) # (batch_size, num_channel, freq_dim, time_step)
sample_output = model(sample_input)
print(sample_output.shape) # (batch_size, num_classes)
torch.Size([2, 4])
def train(model, epoch, loss_fn, optimizer, train_loader):
start_time = time.time()
model = model.train() # set the model to training mode. Always do this before you start training!
train_loss = 0.
true_label = torch.zeros(0)
pred_label = torch.zeros(0)
# load batch data
for batch_idx, data in enumerate(train_loader):
train_batch = data[0].unsqueeze(1)
train_label = data[1]
# add ture label
true_label = torch.cat([true_label, train_label])
# clean up the gradients in the optimizer
# this should be called for each batch
optimizer.zero_grad()
output = model(train_batch)
# add pred label
pred_label = torch.cat([pred_label, torch.argmax(output, axis = 1)])
# loss function as objective
loss = loss_fn(output, train_label)
# automatically calculate the backward pass
loss.backward()
# perform the actual backpropagation
optimizer.step()
train_loss += loss.data.item()
accuracy = (torch.sum(true_label == pred_label).item())/len(true_label)
train_loss /= (batch_idx+1)
print('-' * 99)
print(' | end of training epoch {:3d} | time: {:5.2f}s | Loss {:5.4f} | Accuracy {:5.4f} |'.format(
epoch, (time.time() - start_time), train_loss, accuracy))
return train_loss
def validate(model, epoch, loss_fn, validation_loader, test = False):
start_time = time.time()
model = model.eval() # set the model to evaluation mode. Always do this during validation or test phase!
validation_loss = 0.
true_label = torch.zeros(0)
pred_label = torch.zeros(0)
# load batch data
for batch_idx, data in enumerate(validation_loader):
test_batch = data[0].unsqueeze(1)
test_label = data[1]
# add ture label
true_label = torch.cat([true_label, test_label])
# you don't need to calculate the backward pass and the gradients during validation
# so you can call torch.no_grad() to only calculate the forward pass to save time and memory
with torch.no_grad():
output = model(test_batch)
# add pred label
pred_label = torch.cat([pred_label, torch.argmax(output, axis = 1)])
# loss function as objective
loss = loss_fn(output, test_label)
validation_loss += loss.data.item()
accuracy = (torch.sum(true_label == pred_label).item())/len(true_label)
validation_loss /= (batch_idx+1)
if test:
print(' | end of test epoch {:3d} | time: {:5.2f}s | Loss {:5.4f} | Accuracy {:5.4f} |'.format(
epoch, (time.time() - start_time), validation_loss, accuracy))
print('-' * 99)
else:
print(' | end of validation epoch {:3d} | time: {:5.2f}s | Loss {:5.4f} | Accuracy {:5.4f} |'.format(
epoch, (time.time() - start_time), validation_loss, accuracy))
print('-' * 99)
return validation_loss, accuracy, true_label, pred_label
model = AlexNet()
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
total_epoch = 150 # train the model for 100 epochs
model_save = 'best_SIM_model.pt' # path to save the best validation model
# main function
training_loss = []
validation_loss = []
valid_acc = []
min_valid_loss = torch.inf
epoch_count = 0
for epoch in range(1, total_epoch + 1):
training_loss.append(train(model, epoch, loss_fn, optimizer, train_loader=train_loader))
valid_loss, acc, _, _ = validate(model, epoch, loss_fn, validation_loader=validation_loader)
validation_loss.append(valid_loss)
valid_acc.append(acc)
if training_loss[-1] == np.min(training_loss):
print(' Best training model found.')
if validation_loss[-1] == np.min(validation_loss):
# save current best model on validation set
with open(model_save, 'wb') as f:
torch.save(model.state_dict(), f)
print(' Best validation model found and saved.')
if validation_loss[-1] > min_valid_loss:
min_valid_loss = validation_loss[-1]
epoch_count += 1
if epoch_count >= 10:
print(" The validation loss has not decrease for 10 epoches, so the training stop early.")
print('-' * 99)
---------------------------------------------------------------------------------------------------
| end of training epoch 1 | time: 0.22s | Loss 1.3888 | Accuracy 0.4242 |
| end of validation epoch 1 | time: 0.07s | Loss 1.3609 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 2 | time: 0.20s | Loss 1.3680 | Accuracy 0.4242 |
| end of validation epoch 2 | time: 0.07s | Loss 1.3349 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 3 | time: 0.20s | Loss 1.3526 | Accuracy 0.4242 |
| end of validation epoch 3 | time: 0.07s | Loss 1.3194 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 4 | time: 0.20s | Loss 1.3701 | Accuracy 0.4242 |
| end of validation epoch 4 | time: 0.08s | Loss 1.2809 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 5 | time: 0.20s | Loss 1.2943 | Accuracy 0.4242 |
| end of validation epoch 5 | time: 0.07s | Loss 1.2836 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 6 | time: 0.19s | Loss 1.3170 | Accuracy 0.4242 |
| end of validation epoch 6 | time: 0.07s | Loss 1.2906 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 7 | time: 0.19s | Loss 1.3667 | Accuracy 0.4242 |
| end of validation epoch 7 | time: 0.07s | Loss 1.2410 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 8 | time: 0.19s | Loss 1.2788 | Accuracy 0.4242 |
| end of validation epoch 8 | time: 0.07s | Loss 1.2320 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 9 | time: 0.19s | Loss 1.2746 | Accuracy 0.4242 |
| end of validation epoch 9 | time: 0.07s | Loss 1.2143 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 10 | time: 0.19s | Loss 1.1574 | Accuracy 0.4242 |
| end of validation epoch 10 | time: 0.07s | Loss 1.1568 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 11 | time: 0.19s | Loss 1.1994 | Accuracy 0.4242 |
| end of validation epoch 11 | time: 0.07s | Loss 1.1832 | Accuracy 0.4242 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 12 | time: 0.19s | Loss 1.1604 | Accuracy 0.4242 |
| end of validation epoch 12 | time: 0.07s | Loss 1.0489 | Accuracy 0.4848 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 13 | time: 0.19s | Loss 1.0485 | Accuracy 0.4848 |
| end of validation epoch 13 | time: 0.07s | Loss 0.9227 | Accuracy 0.5455 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 14 | time: 0.19s | Loss 1.0204 | Accuracy 0.4848 |
| end of validation epoch 14 | time: 0.07s | Loss 0.7686 | Accuracy 0.6970 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 15 | time: 0.19s | Loss 0.8902 | Accuracy 0.6364 |
| end of validation epoch 15 | time: 0.07s | Loss 0.6401 | Accuracy 0.7576 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 16 | time: 0.19s | Loss 0.6932 | Accuracy 0.6061 |
| end of validation epoch 16 | time: 0.07s | Loss 0.6402 | Accuracy 0.8182 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 17 | time: 0.19s | Loss 0.6895 | Accuracy 0.6970 |
| end of validation epoch 17 | time: 0.07s | Loss 0.6518 | Accuracy 0.6364 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 18 | time: 0.19s | Loss 0.8590 | Accuracy 0.6061 |
| end of validation epoch 18 | time: 0.07s | Loss 0.6055 | Accuracy 0.6970 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 19 | time: 0.19s | Loss 0.5741 | Accuracy 0.7576 |
| end of validation epoch 19 | time: 0.07s | Loss 0.4485 | Accuracy 0.8182 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 20 | time: 0.19s | Loss 0.5487 | Accuracy 0.7576 |
| end of validation epoch 20 | time: 0.07s | Loss 0.4217 | Accuracy 0.9091 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 21 | time: 0.19s | Loss 0.4472 | Accuracy 0.7576 |
| end of validation epoch 21 | time: 0.07s | Loss 0.3904 | Accuracy 0.8485 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 22 | time: 0.20s | Loss 0.5156 | Accuracy 0.7879 |
| end of validation epoch 22 | time: 0.07s | Loss 0.3138 | Accuracy 0.9091 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 23 | time: 0.19s | Loss 0.4272 | Accuracy 0.8182 |
| end of validation epoch 23 | time: 0.07s | Loss 0.3093 | Accuracy 0.9091 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 24 | time: 0.19s | Loss 0.3741 | Accuracy 0.8485 |
| end of validation epoch 24 | time: 0.07s | Loss 0.1956 | Accuracy 0.8788 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 25 | time: 0.19s | Loss 0.4746 | Accuracy 0.8182 |
| end of validation epoch 25 | time: 0.07s | Loss 0.6316 | Accuracy 0.8485 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 26 | time: 0.19s | Loss 0.9171 | Accuracy 0.7273 |
| end of validation epoch 26 | time: 0.07s | Loss 0.4739 | Accuracy 0.8182 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 27 | time: 0.19s | Loss 0.4414 | Accuracy 0.8182 |
| end of validation epoch 27 | time: 0.07s | Loss 0.3317 | Accuracy 0.8485 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 28 | time: 0.19s | Loss 0.6369 | Accuracy 0.7879 |
| end of validation epoch 28 | time: 0.07s | Loss 0.2065 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 29 | time: 0.19s | Loss 0.4971 | Accuracy 0.8485 |
| end of validation epoch 29 | time: 0.07s | Loss 0.2087 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 30 | time: 0.19s | Loss 0.2964 | Accuracy 0.8485 |
| end of validation epoch 30 | time: 0.07s | Loss 0.2168 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 31 | time: 0.19s | Loss 0.2869 | Accuracy 0.9394 |
| end of validation epoch 31 | time: 0.07s | Loss 0.1767 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 32 | time: 0.19s | Loss 0.1962 | Accuracy 0.8788 |
| end of validation epoch 32 | time: 0.07s | Loss 0.1020 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 33 | time: 0.24s | Loss 0.1301 | Accuracy 0.9697 |
| end of validation epoch 33 | time: 0.07s | Loss 0.1033 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 34 | time: 0.19s | Loss 0.4525 | Accuracy 0.8788 |
| end of validation epoch 34 | time: 0.07s | Loss 0.0913 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 35 | time: 0.19s | Loss 0.3171 | Accuracy 0.9091 |
| end of validation epoch 35 | time: 0.07s | Loss 0.2535 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 36 | time: 0.19s | Loss 0.3460 | Accuracy 0.9394 |
| end of validation epoch 36 | time: 0.07s | Loss 0.1356 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 37 | time: 0.19s | Loss 0.3500 | Accuracy 0.9091 |
| end of validation epoch 37 | time: 0.07s | Loss 0.1709 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 38 | time: 0.19s | Loss 0.2291 | Accuracy 0.8788 |
| end of validation epoch 38 | time: 0.07s | Loss 0.1720 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 39 | time: 0.32s | Loss 0.2210 | Accuracy 0.9091 |
| end of validation epoch 39 | time: 0.07s | Loss 0.1652 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 40 | time: 0.20s | Loss 0.1782 | Accuracy 0.9394 |
| end of validation epoch 40 | time: 0.07s | Loss 0.0835 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 41 | time: 0.19s | Loss 0.0748 | Accuracy 0.9697 |
| end of validation epoch 41 | time: 0.07s | Loss 0.0271 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 42 | time: 0.19s | Loss 0.0458 | Accuracy 1.0000 |
| end of validation epoch 42 | time: 0.07s | Loss 0.0146 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 43 | time: 0.30s | Loss 0.3887 | Accuracy 0.9697 |
| end of validation epoch 43 | time: 0.07s | Loss 0.0070 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 44 | time: 0.19s | Loss 0.3915 | Accuracy 0.9394 |
| end of validation epoch 44 | time: 0.07s | Loss 0.1346 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 45 | time: 0.19s | Loss 0.1783 | Accuracy 0.9394 |
| end of validation epoch 45 | time: 0.07s | Loss 0.1553 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 46 | time: 0.19s | Loss 0.2117 | Accuracy 0.9091 |
| end of validation epoch 46 | time: 0.07s | Loss 0.0588 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 47 | time: 0.19s | Loss 0.1215 | Accuracy 0.9394 |
| end of validation epoch 47 | time: 0.07s | Loss 0.0541 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 48 | time: 0.19s | Loss 0.2608 | Accuracy 0.9394 |
| end of validation epoch 48 | time: 0.07s | Loss 0.0502 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 49 | time: 0.19s | Loss 0.1397 | Accuracy 0.9394 |
| end of validation epoch 49 | time: 0.07s | Loss 0.0402 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 50 | time: 0.19s | Loss 0.3141 | Accuracy 0.9394 |
| end of validation epoch 50 | time: 0.07s | Loss 0.0414 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 51 | time: 0.19s | Loss 0.1004 | Accuracy 0.9697 |
| end of validation epoch 51 | time: 0.07s | Loss 0.0469 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 52 | time: 0.19s | Loss 0.0564 | Accuracy 1.0000 |
| end of validation epoch 52 | time: 0.07s | Loss 0.0137 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 53 | time: 0.26s | Loss 0.0624 | Accuracy 0.9697 |
| end of validation epoch 53 | time: 0.07s | Loss 0.0061 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 54 | time: 0.19s | Loss 0.0570 | Accuracy 0.9697 |
| end of validation epoch 54 | time: 0.07s | Loss 0.0132 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 55 | time: 0.19s | Loss 0.2753 | Accuracy 0.9394 |
| end of validation epoch 55 | time: 0.07s | Loss 0.0064 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 56 | time: 0.30s | Loss 0.0210 | Accuracy 1.0000 |
| end of validation epoch 56 | time: 0.07s | Loss 0.0072 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 57 | time: 0.32s | Loss 0.0178 | Accuracy 1.0000 |
| end of validation epoch 57 | time: 0.07s | Loss 0.0065 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 58 | time: 0.19s | Loss 0.0352 | Accuracy 1.0000 |
| end of validation epoch 58 | time: 0.07s | Loss 0.0038 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 59 | time: 0.23s | Loss 0.0074 | Accuracy 1.0000 |
| end of validation epoch 59 | time: 0.07s | Loss 0.0028 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 60 | time: 0.23s | Loss 0.0272 | Accuracy 1.0000 |
| end of validation epoch 60 | time: 0.07s | Loss 0.0024 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 61 | time: 0.34s | Loss 0.0893 | Accuracy 0.9697 |
| end of validation epoch 61 | time: 0.07s | Loss 0.0052 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 62 | time: 0.29s | Loss 0.0069 | Accuracy 1.0000 |
| end of validation epoch 62 | time: 0.07s | Loss 0.0049 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 63 | time: 0.28s | Loss 0.0147 | Accuracy 1.0000 |
| end of validation epoch 63 | time: 0.07s | Loss 0.0028 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 64 | time: 0.34s | Loss 0.0035 | Accuracy 1.0000 |
| end of validation epoch 64 | time: 0.07s | Loss 0.0012 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 65 | time: 0.19s | Loss 0.0462 | Accuracy 0.9697 |
| end of validation epoch 65 | time: 0.07s | Loss 0.0040 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 66 | time: 0.19s | Loss 0.1457 | Accuracy 0.9697 |
| end of validation epoch 66 | time: 0.07s | Loss 0.0135 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 67 | time: 0.19s | Loss 0.1055 | Accuracy 0.9697 |
| end of validation epoch 67 | time: 0.07s | Loss 0.0367 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 68 | time: 0.32s | Loss 0.2426 | Accuracy 0.9394 |
| end of validation epoch 68 | time: 0.07s | Loss 0.0102 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 69 | time: 0.19s | Loss 0.2893 | Accuracy 0.9091 |
| end of validation epoch 69 | time: 0.07s | Loss 0.0260 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 70 | time: 0.31s | Loss 0.6257 | Accuracy 0.9697 |
| end of validation epoch 70 | time: 0.07s | Loss 0.0166 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 71 | time: 0.19s | Loss 0.6670 | Accuracy 0.8485 |
| end of validation epoch 71 | time: 0.07s | Loss 0.4167 | Accuracy 0.8485 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 72 | time: 0.19s | Loss 0.8130 | Accuracy 0.7576 |
| end of validation epoch 72 | time: 0.07s | Loss 0.5690 | Accuracy 0.8485 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 73 | time: 0.19s | Loss 0.3551 | Accuracy 0.9091 |
| end of validation epoch 73 | time: 0.07s | Loss 0.3071 | Accuracy 0.8788 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 74 | time: 0.19s | Loss 0.3246 | Accuracy 0.8788 |
| end of validation epoch 74 | time: 0.07s | Loss 0.2477 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 75 | time: 0.19s | Loss 0.3055 | Accuracy 0.9697 |
| end of validation epoch 75 | time: 0.07s | Loss 0.2025 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 76 | time: 0.19s | Loss 0.2230 | Accuracy 0.9394 |
| end of validation epoch 76 | time: 0.07s | Loss 0.1404 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 77 | time: 0.27s | Loss 0.1767 | Accuracy 0.9697 |
| end of validation epoch 77 | time: 0.07s | Loss 0.1287 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 78 | time: 0.20s | Loss 0.1201 | Accuracy 0.9697 |
| end of validation epoch 78 | time: 0.07s | Loss 0.1155 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 79 | time: 0.19s | Loss 0.1436 | Accuracy 0.9697 |
| end of validation epoch 79 | time: 0.07s | Loss 0.0988 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 80 | time: 0.23s | Loss 0.1106 | Accuracy 0.9697 |
| end of validation epoch 80 | time: 0.07s | Loss 0.0333 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 81 | time: 0.52s | Loss 0.1867 | Accuracy 0.9697 |
| end of validation epoch 81 | time: 0.07s | Loss 0.0292 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 82 | time: 0.21s | Loss 0.1893 | Accuracy 0.9697 |
| end of validation epoch 82 | time: 0.07s | Loss 0.2886 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 83 | time: 0.20s | Loss 0.6643 | Accuracy 0.9697 |
| end of validation epoch 83 | time: 0.07s | Loss 0.0416 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 84 | time: 0.20s | Loss 0.0409 | Accuracy 1.0000 |
| end of validation epoch 84 | time: 0.07s | Loss 0.0609 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 85 | time: 0.21s | Loss 0.0397 | Accuracy 1.0000 |
| end of validation epoch 85 | time: 0.07s | Loss 0.0301 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 86 | time: 0.42s | Loss 0.0452 | Accuracy 0.9697 |
| end of validation epoch 86 | time: 0.07s | Loss 0.0240 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 87 | time: 0.27s | Loss 0.0338 | Accuracy 1.0000 |
| end of validation epoch 87 | time: 0.07s | Loss 0.0238 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 88 | time: 0.46s | Loss 0.0238 | Accuracy 1.0000 |
| end of validation epoch 88 | time: 0.07s | Loss 0.0297 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 89 | time: 0.19s | Loss 0.0554 | Accuracy 0.9697 |
| end of validation epoch 89 | time: 0.07s | Loss 0.0060 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 90 | time: 0.33s | Loss 0.1680 | Accuracy 0.9697 |
| end of validation epoch 90 | time: 0.07s | Loss 0.0474 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 91 | time: 0.22s | Loss 0.0118 | Accuracy 1.0000 |
| end of validation epoch 91 | time: 0.07s | Loss 0.2501 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 92 | time: 0.24s | Loss 0.1239 | Accuracy 0.9697 |
| end of validation epoch 92 | time: 0.07s | Loss 0.1037 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 93 | time: 0.42s | Loss 0.7356 | Accuracy 0.9697 |
| end of validation epoch 93 | time: 0.07s | Loss 0.7833 | Accuracy 0.7273 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 94 | time: 0.22s | Loss 1.2108 | Accuracy 0.6667 |
| end of validation epoch 94 | time: 0.07s | Loss 0.3200 | Accuracy 0.8788 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 95 | time: 0.20s | Loss 0.6495 | Accuracy 0.9091 |
| end of validation epoch 95 | time: 0.07s | Loss 0.3242 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 96 | time: 0.19s | Loss 0.3275 | Accuracy 0.9394 |
| end of validation epoch 96 | time: 0.07s | Loss 0.2867 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 97 | time: 0.20s | Loss 0.3656 | Accuracy 0.9394 |
| end of validation epoch 97 | time: 0.07s | Loss 0.2194 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 98 | time: 0.20s | Loss 0.2632 | Accuracy 0.9394 |
| end of validation epoch 98 | time: 0.07s | Loss 0.1725 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 99 | time: 0.19s | Loss 0.1142 | Accuracy 0.9697 |
| end of validation epoch 99 | time: 0.07s | Loss 0.1130 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 100 | time: 0.19s | Loss 0.1215 | Accuracy 0.9697 |
| end of validation epoch 100 | time: 0.07s | Loss 0.0825 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 101 | time: 0.20s | Loss 0.2925 | Accuracy 0.8485 |
| end of validation epoch 101 | time: 0.07s | Loss 0.1281 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 102 | time: 0.33s | Loss 0.2043 | Accuracy 0.9394 |
| end of validation epoch 102 | time: 0.07s | Loss 0.1112 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 103 | time: 0.26s | Loss 0.1380 | Accuracy 0.9697 |
| end of validation epoch 103 | time: 0.07s | Loss 0.0814 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 104 | time: 0.28s | Loss 0.1230 | Accuracy 0.9394 |
| end of validation epoch 104 | time: 0.07s | Loss 0.0624 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 105 | time: 0.20s | Loss 0.0733 | Accuracy 0.9697 |
| end of validation epoch 105 | time: 0.07s | Loss 0.0365 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 106 | time: 0.40s | Loss 0.0682 | Accuracy 0.9697 |
| end of validation epoch 106 | time: 0.07s | Loss 0.0120 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 107 | time: 0.20s | Loss 0.2202 | Accuracy 0.9697 |
| end of validation epoch 107 | time: 0.07s | Loss 0.1461 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 108 | time: 0.19s | Loss 0.2348 | Accuracy 0.9394 |
| end of validation epoch 108 | time: 0.07s | Loss 0.2122 | Accuracy 0.9091 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 109 | time: 0.20s | Loss 0.0390 | Accuracy 1.0000 |
| end of validation epoch 109 | time: 0.07s | Loss 0.1171 | Accuracy 0.9394 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 110 | time: 0.20s | Loss 0.1162 | Accuracy 0.9394 |
| end of validation epoch 110 | time: 0.07s | Loss 0.0607 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 111 | time: 0.29s | Loss 0.0783 | Accuracy 0.9697 |
| end of validation epoch 111 | time: 0.07s | Loss 0.0375 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 112 | time: 0.23s | Loss 0.1347 | Accuracy 0.9697 |
| end of validation epoch 112 | time: 0.07s | Loss 0.0198 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 113 | time: 0.19s | Loss 0.0310 | Accuracy 1.0000 |
| end of validation epoch 113 | time: 0.07s | Loss 0.0270 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 114 | time: 0.20s | Loss 0.1434 | Accuracy 0.9394 |
| end of validation epoch 114 | time: 0.07s | Loss 0.0627 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 115 | time: 0.30s | Loss 0.2313 | Accuracy 0.9394 |
| end of validation epoch 115 | time: 0.07s | Loss 0.0126 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 116 | time: 0.20s | Loss 1.2355 | Accuracy 0.8788 |
| end of validation epoch 116 | time: 0.07s | Loss 0.4536 | Accuracy 0.8788 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 117 | time: 0.20s | Loss 0.4257 | Accuracy 0.8182 |
| end of validation epoch 117 | time: 0.07s | Loss 0.0965 | Accuracy 0.9697 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 118 | time: 0.20s | Loss 0.2212 | Accuracy 0.9394 |
| end of validation epoch 118 | time: 0.07s | Loss 0.0607 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 119 | time: 0.20s | Loss 0.1113 | Accuracy 0.9697 |
| end of validation epoch 119 | time: 0.07s | Loss 0.0868 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 120 | time: 0.20s | Loss 0.1022 | Accuracy 0.9697 |
| end of validation epoch 120 | time: 0.07s | Loss 0.0445 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 121 | time: 0.19s | Loss 0.0358 | Accuracy 1.0000 |
| end of validation epoch 121 | time: 0.07s | Loss 0.0303 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 122 | time: 0.20s | Loss 0.0314 | Accuracy 1.0000 |
| end of validation epoch 122 | time: 0.07s | Loss 0.0221 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 123 | time: 0.20s | Loss 0.0271 | Accuracy 1.0000 |
| end of validation epoch 123 | time: 0.07s | Loss 0.0151 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 124 | time: 0.19s | Loss 0.0267 | Accuracy 1.0000 |
| end of validation epoch 124 | time: 0.07s | Loss 0.0107 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 125 | time: 0.20s | Loss 0.0166 | Accuracy 1.0000 |
| end of validation epoch 125 | time: 0.07s | Loss 0.0081 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 126 | time: 0.35s | Loss 0.0173 | Accuracy 1.0000 |
| end of validation epoch 126 | time: 0.07s | Loss 0.0064 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 127 | time: 0.31s | Loss 0.0089 | Accuracy 1.0000 |
| end of validation epoch 127 | time: 0.07s | Loss 0.0036 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 128 | time: 0.20s | Loss 0.0017 | Accuracy 1.0000 |
| end of validation epoch 128 | time: 0.07s | Loss 0.0025 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 129 | time: 0.21s | Loss 0.0792 | Accuracy 0.9697 |
| end of validation epoch 129 | time: 0.07s | Loss 0.0032 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 130 | time: 0.20s | Loss 0.0156 | Accuracy 1.0000 |
| end of validation epoch 130 | time: 0.07s | Loss 0.0081 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 131 | time: 0.35s | Loss 0.0786 | Accuracy 0.9697 |
| end of validation epoch 131 | time: 0.07s | Loss 0.0087 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 132 | time: 0.23s | Loss 0.0065 | Accuracy 1.0000 |
| end of validation epoch 132 | time: 0.07s | Loss 0.0095 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 133 | time: 0.30s | Loss 0.0071 | Accuracy 1.0000 |
| end of validation epoch 133 | time: 0.07s | Loss 0.0083 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 134 | time: 0.32s | Loss 0.0148 | Accuracy 1.0000 |
| end of validation epoch 134 | time: 0.07s | Loss 0.0058 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 135 | time: 0.22s | Loss 0.0165 | Accuracy 1.0000 |
| end of validation epoch 135 | time: 0.07s | Loss 0.0033 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 136 | time: 0.20s | Loss 0.0015 | Accuracy 1.0000 |
| end of validation epoch 136 | time: 0.07s | Loss 0.0018 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 137 | time: 0.21s | Loss 0.0033 | Accuracy 1.0000 |
| end of validation epoch 137 | time: 0.07s | Loss 0.0012 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 138 | time: 0.20s | Loss 0.0042 | Accuracy 1.0000 |
| end of validation epoch 138 | time: 0.07s | Loss 0.0009 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 139 | time: 0.20s | Loss 0.0025 | Accuracy 1.0000 |
| end of validation epoch 139 | time: 0.07s | Loss 0.0008 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 140 | time: 0.34s | Loss 0.0014 | Accuracy 1.0000 |
| end of validation epoch 140 | time: 0.07s | Loss 0.0006 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best training model found.
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 141 | time: 0.32s | Loss 0.0046 | Accuracy 1.0000 |
| end of validation epoch 141 | time: 0.07s | Loss 0.0005 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
Best validation model found and saved.
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 142 | time: 0.32s | Loss 0.0260 | Accuracy 0.9697 |
| end of validation epoch 142 | time: 0.07s | Loss 0.0006 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 143 | time: 0.22s | Loss 0.0032 | Accuracy 1.0000 |
| end of validation epoch 143 | time: 0.07s | Loss 0.0009 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 144 | time: 0.44s | Loss 0.0276 | Accuracy 1.0000 |
| end of validation epoch 144 | time: 0.07s | Loss 0.0008 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 145 | time: 0.29s | Loss 0.1290 | Accuracy 0.9697 |
| end of validation epoch 145 | time: 0.07s | Loss 0.0027 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 146 | time: 0.21s | Loss 0.0239 | Accuracy 1.0000 |
| end of validation epoch 146 | time: 0.07s | Loss 0.0032 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 147 | time: 0.31s | Loss 0.0052 | Accuracy 1.0000 |
| end of validation epoch 147 | time: 0.07s | Loss 0.0050 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 148 | time: 0.30s | Loss 0.0073 | Accuracy 1.0000 |
| end of validation epoch 148 | time: 0.07s | Loss 0.0052 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 149 | time: 0.22s | Loss 0.0101 | Accuracy 1.0000 |
| end of validation epoch 149 | time: 0.07s | Loss 0.0032 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
| end of training epoch 150 | time: 0.23s | Loss 0.0019 | Accuracy 1.0000 |
| end of validation epoch 150 | time: 0.07s | Loss 0.0024 | Accuracy 1.0000 |
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
plt.plot(valid_acc, label='Validation Acc')
plt.xlabel('Epoch')
plt.ylabel("Validation Accuracy")
Text(0, 0.5, 'Validation Accuracy')
model_save = "best_SIM_model.pt"
loss_fn = torch.nn.CrossEntropyLoss()
model.load_state_dict(torch.load(model_save))
validation_loss, accuracy, true_label, pred_label = validate(model, 1, loss_fn, validation_loader)
| end of validation epoch 1 | time: 0.08s | Loss 0.0005 | Accuracy 1.0000 | ---------------------------------------------------------------------------------------------------
speakers = pred_label+1
speakers
tensor([3., 4., 2., 3., 3., 1., 4., 2., 3., 2., 3., 2., 2., 3., 3., 1., 3., 3.,
2., 2., 3., 4., 1., 3., 1., 4., 1., 4., 3., 4., 3., 2., 3.])
from transformers import Wav2Vec2ForCTC, Wav2Vec2Tokenizer
# Load pretrained model and tokenizer
tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'Wav2Vec2CTCTokenizer'. The class this function is called from is 'Wav2Vec2Tokenizer'. e:\Python\lib\site-packages\transformers\models\wav2vec2\tokenization_wav2vec2.py:752: FutureWarning: The class `Wav2Vec2Tokenizer` is deprecated and will be removed in version 5 of Transformers. Please use `Wav2Vec2Processor` or `Wav2Vec2CTCTokenizer` instead. warnings.warn( Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
# map each segment to the audio
merged_spk_seg = {}
time_spk_seg = {}
speakers = np.array(speakers, dtype=int)
for spk in range(len(speakers)):
if speakers[spk] not in merged_spk_seg.keys():
merged_spk_seg[speakers[spk]] = []
merged_spk_seg[speakers[spk]].append(magnitudes[spk])
time_spk_seg[speakers[spk]] = []
time_spk_seg[speakers[spk]].append([starts_time[spk], ends_time[spk]])
else:
merged_spk_seg[speakers[spk]].append(magnitudes[spk])
time_spk_seg[speakers[spk]].append([starts_time[spk], ends_time[spk]])
speakers_pyannote = []
starts_pyannote = []
ends_pyannote = []
transcriptions = []
for spk in time_spk_seg:
for seg in range(len(time_spk_seg[spk])):
# get the index of seg in mag
left, right = time_spk_seg[spk][seg][0], time_spk_seg[spk][seg][1]
# get the sepc with phase
example_audio_spec = audio_spec[:, left:right+1]
# calculate phase
phase = np.angle(example_audio_spec)
# get the segmentation
t_seg = merged_spk_seg[spk][seg][:,:example_audio_spec.shape[1]]
stft_result = t_seg * np.exp(1j * phase)
# make it back to waveform
waveform = librosa.istft(np.asarray(stft_result), win_length=512, hop_length=256)
# tokenizer the input
inputs = tokenizer(waveform, return_tensors="pt", padding="longest", truncation=True)
with torch.no_grad():
# generate output
logits = model(inputs.input_values).logits
# decode output
predicted_ids = torch.argmax(logits, dim=-1)
transcription = tokenizer.batch_decode(predicted_ids)[0]
speakers_pyannote.append(spk)
Pred_start = index_time_transform(left)
Pred_end = index_time_transform(right)
for i in label:
for j in label[i]:
if j == "Audio":
continue
True_start = label[i][j]['Start']
True_end = label[i][j]['End']
if abs(True_start - Pred_start) < 1 and abs(True_end-Pred_end) < 1:
Pred_start = True_start
Pred_end = True_end
starts_pyannote.append(Pred_start)
ends_pyannote.append(Pred_end)
transcriptions.append(transcription)
print(f"Speaker {spk}: {index_time_transform(left)} -> {index_time_transform(right)}, {transcription}")
Speaker 3: 0.0 -> 2.688, YES AND A VERY RESPECTABLE ONE Speaker 3: 8.624 -> 13.856, SHAS TO RAMBLES AND MODOC MEMORIES Speaker 3: 14.96 -> 18.528, THE BOYS BELONG TO THE GENTLEMAN WHO IS A WIDOWER Speaker 3: 27.632 -> 31.888, GO QUIETLY ALONE Speaker 3: 35.2 -> 40.4, DO THEY STILL INSIST THAT MISS CHALLONER WAS THE ONLY PERSON IN THE ROOM WITH THEM AT THIS TIME Speaker 3: 44.48 -> 51.584, ONLY A LITTLE FOOD WILL BE REQUIRED Speaker 3: 52.032 -> 56.944, I KNOW IT SOUNDS FOOLISH BUT THE ALTERNATIVE IS SO IMPROBABLE Speaker 3: 58.192 -> 63.712, THE VERY EMPERORS HAD ARRAYED THEMSELVES ON HER SIDE Speaker 3: 64.512 -> 67.968, I TOOK QUITE A FANCY TO HIM WHY Speaker 3: 71.664 -> 75.12, SEEMED REMARKABLY SMOOTH AND UNBROKEN Speaker 3: 79.104 -> 85.44, BUT IF THE EMPERORS HAD BECOME CHRISTIAN THE EMPIRE HAD NOT Speaker 3: 98.416 -> 103.264, TWO DAYS AFTERWARDS OUR NOCTURNAL ORGY BEGAN TO BE TALKED OF Speaker 3: 103.936 -> 107.664, I ASKED AS SOON AS GEORGE HAD RETURNED TO MY SIDE Speaker 3: 113.808 -> 118.832, HE WANTS ME TO STAND READY TO OBEY ANY SUMMONS THE POLICE MAY SEND ME Speaker 4: 3.488 -> 7.328, OUR SCANDALOUS PROCEEDINGS OFTEN EXPOSED US TO THE GREATEST DANGER Speaker 4: 18.96 -> 26.816, GEORGE NODDED ONE WHO WRITES OF SUCH AN ERA LABORS UNDER A TROUBLESOME DISADVANTAGE Speaker 4: 75.936 -> 78.464, WE TOOK OUR THREE PRISONERS TO A LARGE BOAT Speaker 4: 90.48 -> 92.256, WHERE IS MY HUSBAND Speaker 4: 92.736 -> 96.896, AND THE NEW BLOOD AT THE ERA OF THE STORY WAS AT HAND Speaker 4: 98.416 -> 103.264, WHAT DOES HE WANT Speaker 2: 8.624 -> 13.856, THE LADY IS NOT THE MOTHER OF THE BOYS BUT THEIR AUNT Speaker 2: 27.632 -> 31.888, THE BOYS LOOK QUITE AWAKE ENOUGH IF THE FATHER DOES NOT Speaker 2: 32.624 -> 33.76, O HARM WILL BEFALL YOU Speaker 2: 41.824 -> 43.904, BUT IT IS FAR BETTER TO GO AFOOT Speaker 2: 44.48 -> 51.584, GEORGE HAVE YOU EVER THOUGHT THAT SHE MIGHT HAVE BEEN A SUICIDE Speaker 2: 68.672 -> 71.264, THE LONG GRAY SLOPES LEADING UP TO THE GLACIER Speaker 2: 71.664 -> 75.12, I AM LOOKING AT HIM NOW Speaker 2: 108.704 -> 113.008, OUNT BREMMER IS THE MOST NOTED STRONGHOLD OF THE SHEEP IN THE WHOLE SHASTA REGION Speaker 1: 18.96 -> 26.816, GEORGE NODDED THIS IS THE AMUSING ADVENTURE WHICH CLOSED OUR EXPLOITS Speaker 1: 58.192 -> 63.712, I INQUIRED OF GEORGE WITH MY EYES STILL ON THIS FURTIVE WATCHER Speaker 1: 79.104 -> 85.44, YES HE'S MERE CURIAL IN ALL HIS MOVEMENTS Speaker 1: 86.064 -> 89.728, THE MAN'S SANA MUST HAVE A CORPUSE SANUM TO INHABIT Speaker 1: 92.736 -> 96.896, NEVER FEAR YOU SHALL SEE HIM AGAIN TO MORROW
pred_reference_pyannote = Annotation()
for i in range(len(speakers_pyannote)):
spk = speakers_pyannote[i]
start_time = starts_pyannote[i]
end_time = ends_pyannote[i]
pred_reference_pyannote[Segment(start_time,end_time)] = spk
pred_reference_pyannote
One problem with this timeline just show the rough segment time this sentence fall into. The overlap of speeches overlap on each other in this graph since they are think as the same segment so they have same start and end time. It could make it looks better if we make it more precise on the specific time for each segment of each spk for one utterance.